Papers with Llama architecture
Small Language Models Also Work With Small Vocabularies: Probing the Linguistic Abilities of Grapheme- and Phoneme-Based Baby Llamas (2025.coling-main)
Copied to clipboard
| Challenge: | Existing studies on LMs have focused on linguistic generalizations and representations from developmentally plausible data. |
| Approach: | They propose to use phoneme- and grapheme-based language models to learn linguistic units at and below the word level. |
| Outcome: | The proposed models can achieve strong performance on syntactic and novel benchmarks and match grapheme-based models in standard tasks and novel evaluations. |
Learning to Look at the Other Side: A Semantic Probing Study of Word Embeddings in LLMs with Enabled Bidirectional Attention (2025.acl-long)
Copied to clipboard
| Challenge: | Autoregressive Large Language Models (LLMs) demonstrate exceptional performance in language understanding and generation tasks, but their application in text embedding tasks has been relatively slow due to the constraints of the unidirectional attention mechanism. |
| Approach: | They propose to enable bidirectional attention in LLMs to improve their ability to represent subsequent context but weaken their utilization of preceding context. |
| Outcome: | The proposed model improves the LLMs’ ability to represent subsequent context but weakens their utilization of preceding context, while contrastive learning training can help to maintain both abilities. |